Method of detecting obstacle around vehicle

ABSTRACT

Disclosed is a method of detecting obstacle around a vehicle. The method of detecting an obstacle around a vehicle, includes: acquiring an image of the obstacle around the vehicle using a monocular camera; creating, by a controller, a distance based cost map, a color based cost map and an edge based cost map from the image; and integrating, by the controller, the distance based cost map, the color based cost map, and the edge based cost map to create a final cost map, and estimating, by the controller, a height of the obstacle from the final cost map.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of Korean PatentApplication No. 10-2017-0050982, filed on Apr. 20, 2017, which isincorporated herein by reference in its entirety.

FIELD

The present disclosure relates to a vehicle, and more particularly to anautonomous vehicle.

BACKGROUND

The statements in this section merely provide background informationrelated to the present disclosure and may not constitute prior art.

An autonomous vehicle travels to a destination by itself even if adriver does not operate a steering wheel, an accelerator pedal, or abrake pedal. The autonomous vehicle is actually different from anunmanned vehicle traveling without a driver, but in these days, theterms “autonomous vehicle” and “unmanned vehicle” are used withoutdistinction.

An example of autonomous driving is automatic parking. The automaticparking means that the autonomous vehicle itself finds empty parkingspace to perform parking by itself. To achieve the automatic parking,the autonomous vehicle should be equipped with three-dimensionalenvironment recognition technology to detect surrounding geographicfeatures and objects.

In order to detect geographical features and objects, a detectiondevice, such as a binocular stereo camera, a radar, a lidar, and anultrasonic sensor, is needed. However, the binocular stereo camera, theradar, and the lidar are expensive. Also, the ultrasonic sensor hasdifficulties in acquiring additional information, and has a shortdetection range.

SUMMARY

The present disclosure provides improved three-dimensional environmentrecognition technology using a single monocular camera of low cost.

In one form of the present disclosure, a method of detecting an obstaclearound a vehicle, includes: acquiring an image of the obstacle aroundthe vehicle using a monocular camera; creating, by a controller, adistance based cost map, a color based cost map and an edge based costmap from the image; and integrating, by the controller, the distancebased cost map, the color based cost map, and the edge based cost map togenerate a final cost map, and estimating, by the controller, a heightof the obstacle from the final cost map.

The creating the distance based cost map includes: generating aplurality of Delaunay triangles by connecting feature points in theimage and performing interpolation of distance information using a planeequation generated by three vertexes of each of the plurality of theDelaunay triangles; and estimating a disparity of pixels included ineach of the plurality of the Delaunay triangles from a result of theinterpolation.

The creating the color based cost map, includes: setting an area wherecolor similarities are measured in the image, measuring the colorsimilarities between all feature points existing in the area, andselecting a greatest color similarity value as a final color similarity;and calculating a difference between sums of the color similarities fromthe measured color similarities to create the color based cost map.

The creating of the edge based cost map, includes: performing edgedetection on the image; and performing distance transformation based onedge detection such that pixels located closer to an edge have lowervalues.

The final cost map is calculated by Equation 6:

c _(t) =w _(d) c _(d) +w _(c) c _(c) +w _(e) c _(e)  <Equation 6>

-   -   where, w_(d) is a weight of the distance cost map, w_(c) is a        weight of the color cost map, and w_(e) is a weight of the edge        cost map, c_(d), color c_(c), and edge c_(e).

In accordance with another aspect of the present disclosure, the methodof detecting an obstacle around a vehicle, includes: acquiring an imageof the obstacle around the vehicle using a monocular camera;reconstructing, by a controller, three-dimensional positions ofcorresponding points in the image; integrating, by the controller,previously reconstructed three-dimensional corresponding points with thecurrently reconstructed three-dimensional corresponding points based ona relative positional relationship between the previously reconstructedthree-dimensional corresponding points and the currently reconstructedthree-dimensional corresponding points; calculating, by the controller,a disparity value by applying a virtual baseline value formed by amovement of the monocular camera to a depth value obtained throughthree-dimensional reconstruction of the corresponding points; andestimating, by the controller, a boundary of the obstacle based on thedisparity value.

The disparity value is calculated using Equation 1:

$\begin{matrix}{{d = {f\frac{B}{Z}}},} & {\text{<}{Equation}\mspace{14mu} 1\text{>}}\end{matrix}$

where the d is the disparity value to be obtained, the B is the virtualbaseline value of the monocular camera, the Z is the depth valueobtained through three-dimensional reconstruction, and the f is a focallength of the monocular camera.

The method further includes changing, when an angle of view of themonocular camera is a wide angle that is greater than or equal to apredetermined angle of view, a u-axis to an incident angular axis θuthrough the following equation:

${\theta_{u} = {{atan}\left( \frac{u - o_{x}}{f} \right)}},$

-   -   where, the u is the value of the u-axis, the 0_(x) is the center        point of the monocular camera, and the f is the focal length of        the monocular camera.

In accordance with another aspect of the present disclosure, a method ofdetecting an obstacle around a vehicle, includes: acquiring an image ofthe obstacle around the vehicle using a monocular camera;reconstructing, by a controller, three-dimensional positions ofcorresponding points in the image, calculating, by the controller, adisparity value by applying a virtual baseline value formed by amovement of the monocular camera to a depth value obtained throughthree-dimensional reconstruction of the corresponding points, andestimating, by the controller, a boundary of the obstacle based on thedisparity value; and creating, by the controller, a distance based costmap, a color based cost map and an edge based cost map from the image,and estimating, by the controller, a height of the obstacle using thedistance based cost map, the color based cost map, and the edge basedcost map.

The disparity value is calculated using Equation 1:

${d = {f\frac{B}{Z}}},$

where the ‘d’ is the disparity value to be obtained, the B is thevirtual baseline value of the monocular camera, the Z is the depth valueobtained through three-dimensional reconstruction, and the f is a focallength of the monocular camera.

The method further includes changing, when an angle of view of themonocular camera is a wide angle that is greater than or equal to apredetermined angle of view, a u-axis to an incident angular axis θuthrough Equation 2:

${\theta_{u} = {{atan}\left( \frac{u - o_{x}}{f} \right)}},$

where the u is the value of the u-axis, the 0x is the center point ofthe monocular camera, and the f is the focal length of the monocularcamera.

The creating of the distance based cost map includes: generating aplurality of Delaunay triangles by connecting feature points of theimage and performing interpolation of distance information using a planeequation generated by three vertexes of each of the plurality of theDelaunay triangles; and estimating a disparity of pixels included ineach of the plurality of the Delaunay triangles from a result of theinterpolation.

The creating the color based cost map, includes: setting an area wherecolor similarity measurement is to be performed in the image, measuringcolor similarities between all feature points existing in the area, andselecting a greatest color similarity value as a final color similarity;and creating the color based cost map by calculating a differencebetween sums of the color similarities from the measured colorsimilarities.

The creating the edge based cost map, includes: performing edgedetection on the image; and performing distance transform on a result ofthe edge detection such that pixels located closer to edges have lowervalues.

The final cost map is calculated by Equation 6:

c _(t) =w _(d) c _(d) +w _(c) c _(c) +w _(e) c _(e),  <Equation 6>

where w_(d) is a weight of the distance cost map, w_(c) is a weight ofthe color cost map, and w_(e) is a weight of the edge cost map.

In accordance with another aspect of the present disclosure, a method ofdetecting an obstacle around a vehicle, includes: acquiring an image ofthe obstacle around the vehicle using a monocular camera; andreconstructing, by a controller, three-dimensional positions ofcorresponding points in the image, calculating, by the controller, adisparity value by applying a virtual baseline value formed by amovement of the monocular camera to a depth value obtained through thethree-dimensional reconstruction of the corresponding points, andestimating, by the controller, a boundary of the obstacle based on thedisparity value.

In another form of the present disclosure, a method of detecting anobstacle around a vehicle, includes: acquiring an image of the obstaclearound the vehicle using a monocular camera; and creating, by acontroller, a distance based cost map, a color based cost map, and anedge based cost map from the image, and estimating, by the controller, aheight of the obstacle using the distance based cost map, the colorbased cost map, and the edge based cost map.

Further areas of applicability will become apparent from the descriptionprovided herein. It should be understood that the description andspecific examples are intended for purposes of illustration only and arenot intended to limit the scope of the present disclosure.

DRAWINGS

In order that the disclosure may be well understood, there will now bedescribed various forms thereof, given by way of example, referencebeing made to the accompanying drawings, in which:

FIG. 1 shows a vehicle;

FIGS. 2A-2B show an obstacle boundary and an obstacle height detected bya monocular camera during autonomous driving of a vehicle;

FIG. 3 shows a control system of the vehicle;

FIG. 4 shows a control method of a vehicle;

FIG. 5 shows a flowchart for a method of obstacle boundary detection;

FIG. 6 shows a flowchart for a method of obstacle height detection;

FIG. 7 shows key frames and their intervals;

FIGS. 8A-8B show the relationship of the corresponding points between apair of key frames;

FIGS. 9A-9B show removal and selection of the three-dimensional points;

FIG. 10 shows a transform of the previously reconstructedthree-dimensional points and integrating the previously reconstructedthree-dimensional points with the currently reconstructedthree-dimensional points;

FIGS. 11A-11B show a integration of a plurality of three-dimensionalreconstruction results acquired with time intervals;

FIGS. 12A-12B show the results of obstacle boundary detection before andafter performing temporal integration of three-dimensional points;

FIGS. 13A-13B show the u-axis range of the corresponding points and thenumber of disparities accumulated per u-axis;

FIG. 14 shows the u-disparity map, the associated image, and thecorresponding points generated;

FIG. 15 shows an optimal path detection result based on the dynamicprogramming (DP) using the cost function;

FIG. 16 shows a valid boundary region of the entire obstacle boundary;

FIGS. 17A-17B show distortion removal through image conversion;

FIGS. 18A-18C show interpolation through creation of Delaunay triangles;

FIGS. 19A-19C show calculation of disparity similarity and a resultthereof;

FIG. 20 shows setting an area of interest and measuring disparity toobtain color based obstacle height information;

FIGS. 21A-21C show a calculation process of the color based cost mapbased on the color similarity calculation;

FIGS. 22A-22B show calculation of an edge based cost map based on edgedetection;

FIG. 23 shows a result of applying a cost map integration process, adynamic programming application, and a final obstacle height estimationresult to an actual image; and

FIG. 24 shows the final result of estimating the boundary and height ofthe obstacle 202 through a series of processes.

The drawings described herein are for illustration purposes only and arenot intended to limit the scope of the present disclosure in any way.

DETAILED DESCRIPTION

The following description is merely exemplary in nature and is notintended to limit the present disclosure, application, or uses. Itshould be understood that throughout the drawings, correspondingreference numerals indicate like or corresponding parts and features.

In the description of the present disclosure, drawings and forms shownin the drawings are exemplary examples of the disclosed presentdisclosure, and there can be various modifications that can replace theforms and the drawings of the present disclosure at the time of filingof the present disclosure.

FIG. 1 is shows a vehicle in one form of the present disclosure. Avehicle 100 shown in FIG. 1 may be an autonomous vehicle. At the rear ofthe vehicle 100 shown in FIG. 1, a monocular camera 102 for recognizinga three-dimensional (3D) environment around the vehicle 100 may bemounted. The monocular camera 102 may be equipped with a fisheye lenshaving an angle of view of 180 degrees.

The monocular camera 102 of the vehicle 100 may be a rear camera forphotographing a rear view of the vehicle 100 when the vehicle 100reverses. Alternatively, the monocular camera 102 of the vehicle 100 maybe a camera mounted at the rear portion of the vehicle 100 among aplurality of cameras respectively mounted at the front, rear, left, andright portions of the vehicle 100 to create around view images of thevehicle 100.

An image photographed by the monocular camera 102 of the vehicle 100 maybe image-processed by a controller (see 302 of FIG. 3), and thendisplayed on a display (for example, a navigation screen, see 316 ofFIG. 3) of the vehicle 100.

FIGS. 2A-2B show an obstacle boundary line and an obstacle heightdetected by the monocular camera during autonomous driving of thevehicle in one form of the present disclosure. The obstacle means anobject that the vehicle 100 should avoid not to collide with it upontraveling. The obstacle may include another vehicle, a geographicalfeature, or a structure existing around the vehicle 100. When themonocular camera 102 of the vehicle 100 is used for autonomous drivingas shown in FIG. 2A, operation in which the vehicle 100 detectssurrounding obstacles 202 using the monocular camera 102 may beoperation of estimating boundaries and heights of the obstacles 202.That is, as shown in FIG. 2B, the vehicle 100 may estimate a boundaryline of the obstacles 202 and heights of the obstacles 202 to therebydistinguish empty parking space 204 from the obstacles 202.

FIG. 3 is a block diagram illustrating control system of the vehicle inone form of the present disclosure. Specifically, the control systemshown in FIG. 3 is to acquire information about an obstacle 202 aroundthe vehicle 100 from an image photographed by the monocular camera 102,and to use the acquired information for autonomous driving.

The controller 302 may control overall operations related to driving ofthe vehicle 100. The controller 302 may be configured with a singleElectronic Control Unit (ECU) or a plurality of ECUs interworking witheach other. The monocular camera 102, a driving state detector 306, anda path controller 308 may be connected to input terminals of thecontroller 302 in such a way to communicate with each other. Powersystem 310, steering system 312, braking system 314, and a display 316may be connected to output terminals of the controller 302 in such a wayto communicate with each other.

The monocular camera 102 may photograph an around view (particularly, arear view) of the vehicle 100 to create an image, as described abovewith reference to FIGS. 1 and 2A-2B. The image photographed by themonocular camera 102 may be transferred to the controller 302, andsubjected to a series of image processing in one form of the presentdisclosure so as to be used to detect a status of an obstacle 202 aroundthe vehicle 100.

The driving state detector 306 may include a plurality of sensors (forexample, a vehicle speed sensor, a steering angle sensor, a yaw ratesensor, and an acceleration sensor) for detecting the driving state ofthe vehicle 100. The driving state detector 306 may detect a drivingstate of the vehicle 100, and transmit information on the detecteddriving state to the controller 302.

The path controller 308 may estimate a current position of the vehicle100, create a path plan about a path along which the vehicle 100 moves,and transmit the path plan to the controller 302.

The power system 310 may include an engine and a transmission of thevehicle 100. The power system 310 may generate power from the engine,and transmit the power to the wheels so that the vehicle 100 can run.

The steering system 312 may use various kinds of sensors and controldevices to provide steering stability of the vehicle 100 with respect toa steering input from a driver. For this, the steering system 312 mayinclude Electronic Power Steering (EPS), Motor Driven Power Steering(MDPS), Active Front Steering (AFS), etc.

The braking system 314 may be used to reduce the speed of the vehicle100 to stop the vehicle 100. The braking system 314 may put a brake onthe vehicle 100 in response to a control signal from the controller 302.For this, the braking system 314 may include Anti-lock Brake System(ABS) and Electronic Stability Control (ECS) System.

The display 316 may be used to display a driving state of the vehicle100 and information desired for driving of the vehicle 100. The display316 may be an independently mounted display or a display attached to amultimedia device (e.g., Audio Video Navigation (AVN) system). On thescreen of the display 316, an around view image of the vehicle 100,photographed by the monocular camera 102, or information provided forautonomous driving may be displayed.

FIG. 4 is a flowchart illustrating a method of controlling the vehiclein one form of the present disclosure. According to the control methodshown in FIG. 4, an around view image of the vehicle 100 may bephotographed using the monocular camera 102, an obstacle 202 existingaround the vehicle 100 may be detected from the around view image, andan image of the obstacle 202 may be displayed on the display 316. Thevehicle 100 may detect the obstacle 202 existing around the vehicle 100using only the monocular camera 102, instead of a binocular stereocamera, a radar, a lidar, and an ultrasonic sensor. The form of thepresent disclosure proposes a method of detecting a boundary of anobstacle and a method of estimating a height of an obstacle, includingnew features.

As shown in FIG. 4, the controller 302 may receive an image from themonocular camera 102 (in operation 402), detect a boundary and a heightof the obstacle from the received image (in operations 404 and 406), anddisplay information of the detected boundary and height of the obstacleon the display 316 (in operation 408).

Operation 404 of detecting the boundary of the obstacle may include:temporally integrating three-dimensional information in order toovercome limitations of non-dense three-dimensional information;creating a u-disparity efficiently based on angular information in awide-angle condition; and generating a displacement value using avirtual baseline when there is no disparity information.

Operation 406 of detecting the height of the obstacle may include:estimating the height of the obstacle by merging distance information,color information, and edge information in order to overcome thelimitations of non-dense three-dimensional information; and increasingthe efficiency of calculation by forcing the obstacle to standvertically on the image through a cylinder transform process in an imagedistortion condition.

Hereinafter, operation 404 of detecting the boundary of the obstacle andoperation of detecting the height of the obstacle in the vehicle 100 inone form of the present disclosure will be described in detail withreference to FIGS. 5 to 24. FIG. 5 is a flowchart illustrating operation404 of detecting the boundary of the obstacle, in detail, and FIG. 6 isa flowchart illustrating operation 406 of detecting the height of theobstacle, in detail. FIGS. 7 to 24 are views for assisting descriptionsrelated to FIGS. 5 and 6.

As shown in FIG. 5, in order to detect the boundary of the obstacle, thecontroller 302 may set an interval of key frames for acquiringcorresponding points from the image photographed by the monocular camera102 (in operation 502). The interval of key frames means an intervalbetween a pair of frames to be used for three-dimensionalreconstruction. FIG. 7 is a diagram illustrating key frames and theirintervals. As shown in FIG. 7, a pair of frames having an appropriateinterval may be selected from among frames of the image photographed bythe monocular camera 102, and corresponding points for 3D reconstructionon the selected pair of frames may be acquired. If the interval betweenthe key frames is too small, or if corresponding points are obtained forall frames without any interval, three-dimensional reconstruction errorsand an amount of data processing may increase excessively although thecorresponding points can be easily acquired. In contrast, if theinterval between key frames is too great, it may be difficult to acquirecorresponding points although three-dimensional reconstruction errorsand an amount of data processing are reduced. Therefore, it is desirableto set an appropriate interval of key frames through a tradeoff betweenthree-dimensional reconstruction errors and ease in acquiringcorresponding points.

Returning to operation 404 of FIG. 5, the controller 302 may acquirecorresponding points between the key frames (in operation 504). Arelationship of corresponding points between a pair of key frames isshown in FIGS. 8A-8B. As shown in FIGS. 8A-8B, corresponding pointsindicate corresponding locations (the same locations on the real-worldcoordinates) in two key frames forming a pair. The controller 302 mayrestore three-dimensional information using a difference in location ofthe corresponding points in the two key frames. A Kanade-Lucas Tomasi(KLT) method may be used to acquire and track a large number of featurepoints from the surface of a road and the surfaces of vehicles (othervehicles photographed by the monocular camera 102) having a relativelysmall number of feature points. However, another method for acquiringcorresponding points, instead of the KLT method, may be used.

Returning to operation 404 of FIG. 5, the controller 302 may estimate amovement of the monocular camera 102, in operation 506. Since themonocular camera 102 installed in the vehicle 100 moves together along atraveling trajectory of the vehicle 100, it may be desired to estimate amovement motion of the monocular camera 102 in order to accuratelyreconstruct a three-dimensional image. In order to estimate a movementof the monocular camera 102, fundamental matrix calculation may need tobe performed based on the corresponding points. For the fundamentalmatrix calculation, a M-estimator Sample Consensus (MSAC) method may beused. In the MSAC method, a fundamental matrix is decomposed to estimatea movement of the monocular camera 102.

Also, the controller 302 may perform filtering on the correspondingpoints in order to remove corresponding points expected to generate alarge error upon three-dimensional reconstruction, in operation 508. Inthe form of the present disclosure, the controller 302 may performfiltering on the corresponding points in various manners. For example,in a method of filtering corresponding points based on a fundamentalmatrix, corresponding points falling within a predetermined distance toepipolar lines are selected, and the remaining corresponding points areremoved. According to another example, in a method of filteringcorresponding points based on de-rotation, corresponding points of whichlengths are longer than or equal to a predetermined length when theinfluence of rotation is excluded are selected, and the remainingcorresponding points are removed.

Through the above-described process, the correspondence points, themovement of the monocular camera 102, and the internal variables of themonocular camera 102 may be acquired. The controller 302 may calculateand reconstruct three-dimensional locations of the corresponding pointsthrough triangulation based on the corresponding points, the movement ofthe monocular camera 102, and the internal variables of the monocularcamera 102, in operation 510. Since three-dimensional pointsreconstructed using only images have no unit, the controller 302 mayreconstruct the three-dimensional points in meters by using a movementdistance of the monocular camera 102 acquired based on the odometry ofthe vehicle 100. However, since the area where the obstacle 202 likelyto collide with the vehicle 100 exists is a main area of interest, asshown in FIGS. 9A-9B, three-dimensional points reconstructed from theground may be removed, and three-dimensional points existing within apredetermined height range (for example, 25 cm to 200 cm) from theground may be selected and reconstructed.

Returning to operation 404 of FIG. 5, the controller 302 may temporallyintegrate the reconstructed three-dimensional points, in operation 512.In order to secure dense three-dimensional shape information, it mayperform three-dimensional reconstruction at predetermined time intervalsand integrate the results of the reconstruction. In another from, thethree-dimensional integration may be performed by reflecting both themovement of the monocular camera 102 and a movement distance acquiredbased on the odometry of the vehicle 100.

That is, as shown in FIG. 10, the controller 302 may acquire a relativepositional relationship (rotational and positional transformation)between the previously (past) reconstructed three-dimensional points andthe currently reconstructed three-dimensional points by applying amovement of the monocular camera 102 and a movement distance acquiredbased on the odometry of the vehicle 100, and then transform thepreviously reconstructed three-dimensional points to integrate thepreviously reconstructed three-dimensional points with the currentlyreconstructed three-dimensional points.

FIGS. 11A-11B show a graph obtained by integrating a plurality ofthree-dimensional reconstruction results acquired with time intervals.In the graph shown in FIG. 11B, points having different brightness arethree-dimensional points reconstructed at different time. As shown inFIGS. 11A-11B, denser three-dimensional points could be obtained bytemporal integration.

FIGS. 12A-12B are diagrams illustrating the results of obstacle boundarydetection before and after performing temporal integration onthree-dimensional points. As shown in FIGS. 12A-12B, obstacle boundariesthat have not been detected before performing temporal integration onthree-dimensional points are correctly detected after performingtemporal integration on the three-dimensional points.

Returning to operation 404 of FIG. 5, the controller 302 may create au-disparity map using the following method, in operation 514. In aconventional stereo (binocular) camera, a u-disparity is generated usinga dense disparity map, and Dynamic Programming (DP) is applied to theu-disparity to detect a boundary of an obstacle. However, when thesingle monocular camera 102 is used, like in the form of the presentdisclosure, it is difficult to use the method used in the conventionalstereo (binocular) camera as it is. Therefore, in the form of thepresent disclosure, two new methods are proposed in order to obtain thesame obstacle detection effect as that obtained through the conventionalstereo (binocular) camera using only the monocular camera 102.

One of the two proposed methods is to calculate a disparity value from amotion stereo result. The disparity means a difference in distance onthe horizontal axis between corresponding points of a pair of frames inan image of a rectified stereo camera. Unlike the case of the stereocamera, no disparity value can be calculated in the case of a motionstereo based on the monocular camera 102, so that a u-disparity cannotbe generated. Therefore, in the case of using the monocular camera 102as in the form of the present disclosure, a virtual baseline B may beapplied to the depth value (Z) obtained by three-dimensionalreconstruction to calculate a u-disparity value d according to Equation1 below.

$\begin{matrix}{d = {f\frac{B}{Z}}} & {\text{<}{Equation}\mspace{14mu} 1\text{>}}\end{matrix}$

In Equation 1, d is a disparity value to be obtained, B is a virtualbaseline value of the monocular camera 102, Z is a depth value obtainedthrough three-dimensional reconstruction, and f is a focal length of themonocular camera 102.

The other one of the proposed two methods is to generate a u-disparitysuitable for a wide-angle condition. In the monocular camera 102, if awide-angle lens such as a fisheye lens is adopted so that an angle ofview is a wide angle, the range of the u-axis (the abscissa of an image)is very wide due to perspective projection. The wide u-axis greatlyincreases the u-disparity, which greatly increases memory usage and anamount of computation of the controller 302. Also, if the u-disparity isvery large, disparities acquired by the fisheye lens are notsufficiently accumulated.

FIGS. 13A-13B show the u-axis range (the upper part of FIG. 13) of thecorresponding points) and the number of disparities accumulated peru-axis (lower part of the FIG. 13). As shown in FIG. 13B, if the u-axisrange is very wide, disparities accumulated per u-axis are very small.In the form of the present disclosure, the u-axis is changed to anincidence angular axis θ_(u) in order to solve this problem. At thistime, θ is generated by dividing the resolution between −90 degrees and+90 degrees in unit of 1 degree. That is, when the angle of view is 180degrees at the right and left sides of the center point of the monocularcamera 102, the angle of view of 180 degrees is divided by 1 degree intoflabellate shape, and an incident angle axis θ_(u) for each angle θ iscalculated. The incident angle axis θ_(u) may be obtained according toEquation 2 below.

$\begin{matrix}{\theta_{u} = {{atan}\left( \frac{u - o_{x}}{f} \right)}} & {\text{<}{Equation}\mspace{14mu} 2\text{>}}\end{matrix}$

In Equation 2, u is a value of the u-axis, 0_(x) is the center point ofthe monocular camera 102, and f is a focal length of the monocularcamera 102.

The u-disparity map is shown in FIG. 14. FIG. 14 shows a u-disparity mapcreated in one form of the present disclosure, an image related to theu-disparity map, and corresponding points.

Returning to operation 404 of FIG. 5, the controller 302 may performoperation of detecting an optimal path based on DP using a cost functionC, as expressed in Equations 3 to 5 below, in order to estimate aspatially smooth obstacle boundary in the u-disparity obtained throughthe above-described method in the form of the present disclosure, inoperation 516.

$\begin{matrix}{{C\left( {\theta,d_{0},d_{1}} \right)} = {{E_{D}\left( {\theta,d_{0}} \right)} + {E_{S}\left( {d_{0},d_{1}} \right)}}} & {\text{<}{Equation}\mspace{14mu} 3\text{>}} \\{{E_{D}\left( {\theta,d_{0}} \right)} = {- {L\left( {\theta,d_{0}} \right)}}} & {\text{<}{Equation}\mspace{14mu} 4\text{>}} \\{{E_{S}\left( {d_{0},d_{1}} \right)} = \left\{ \begin{matrix}{{{- c_{s}}{l\left( {d_{0},d_{1}} \right)}},} & {{{if}\mspace{14mu} {l\left( {d_{0},d_{1}} \right)}} < T_{s}} \\{{{- c_{s}}T_{s}},} & {{{if}\mspace{14mu} {l\left( {d_{0},d_{1}} \right)}} \geq T_{s}}\end{matrix} \right.} & {\text{<}{Equation}\mspace{14mu} 5\text{>}}\end{matrix}$

In Equations 3 to 5, θ is any arbitrary angle within the angle of viewof the monocular camera 102, d₀ and d₁ are disparities, E_(D) is a dataterm, E_(S) is a smoothness term, L(θ, d₀) is an accumulated value of alocation (θ, d₀) of the u-disparity, I(d₀, d₁) is an absolute value of adifference in distance on the Z axis between d₀ and d₁, c_(S) is aweight of the smoothness term, and T_(S) is a maximum value of thedifference in distance on the Z axis.

FIG. 15 shows a result obtained by detecting an optimal path based on DPusing the cost function expressed in Equations 3 to 5. In FIG. 15, theupper figure shows an obstacle boundary detected from a u-disparity, andthe lower figure shows an image on which the obstacle boundary isrepresented.

Returning to FIG. 5, the controller 302 may extract a valid obstacleregion from the detected obstacle boundary, in operation 518. Anobstacle boundary detected based on DP includes not only theaccumulation of three-dimensional points, but also locations connectedand detected when a path is created based on DP. Therefore, the detectedobstacle boundary can be divided into a valid region and an invalidregion.

For this reason, the controller 302 may classify positions at which theresult of accumulation exists on a path of u-disparities as validobstacle boundaries, and classify positions at which no result ofaccumulation exists as invalid obstacle boundaries. FIG. 16 is a diagramillustrating a valid boundary region of the entire obstacle boundary. InFIG. 16, regions denoted by a reference numeral 1602 are obstacleboundaries detected based on DP, and regions denoted by a referencenumeral 1604 are valid obstacle boundaries.

FIG. 6 is a flowchart illustrating operation 406 of detecting a heightof an obstacle in the vehicle in the form of the present disclosure. Inoperation 406 of detecting the height of the obstacle as shown in FIG.6, the controller 302 may create three cost maps, and integrate thethree cost maps to estimate the height of the obstacle.

As first operation for detecting the height of the obstacle, thecontroller 302 may convert an image photographed by the monocular camera102, in operation 604. FIGS. 17A-17B are views for describing distortionremoval through image conversion in the form of the present disclosure.Through the image conversion, distortion is removed from an imagedistorted by a wide-angle lens (for example, a fish-eye lens) as shownin FIG. 17A so that the image is converted into an image shown in FIG.17B. The controller 302 may reduce an amount of computation byperforming image conversion under an assumption that the obstacle 202 iserected along the vertical axis of the image. However, it is impracticalto use the assumption that the obstacle 202 is erected along thevertical axis of the image if the monocular camera 102 is inclined or ifdistortion exists by the fish-eye lens. Accordingly, the aboveassumption may be made by compensating the angle of the monocular camera102 and performing image conversion based on a cylindrical projectionmethod. As shown in FIGS. 17A-17B, obstacles appearing on the imageafter the image conversion stand in the same direction as the verticalaxis of the image.

Returning to operation 406 of FIG. 6, the controller 302 may createthree cost maps from the converted image, in operations 606, 608, and610. In the form of the present disclosure, the controller 302 maycreate three cost maps using three methods: a method 606 of creating acost map based on distance; a method 608 of creating a cost map based oncolor; and a method 610 of creating a cost map based on edge.

First, the controller 302 may create a distance based cost map from theconverted image, in operation 606. Distance information acquired througha motion stereo based on the monocular camera 102 has a limitation ofinsufficient density to estimate the height of the obstacle 202.Therefore, in the form of the present disclosure, interpolation may beperformed on acquired distance information to obtain accurate distanceinformation.

The controller 302 may create Delaunay triangles as shown in FIGS.18A-18C based on feature points, and then perform interpolation ofdistance information using a plane Equation formed by three vertexes ofthe Delaunay triangles. FIGS. 18A-18C are a diagram illustratinginterpolation through creation of Delaunay triangles in the form of thepresent disclosure. In each of the Delaunay triangles shown in FIGS.18A-18C, three-dimensional coordinates of each vertex are x and ycoordinates and disparity values of the image. The controller 302 mayestimate disparities of all pixels in the Delaunay triangles afteracquiring the plane Equation of the Delaunay triangles. The controller302 can obtain dense disparities by applying the above-describedprocedure to all the Delaunay triangles.

The disparity of the obstacle boundary may need to be similar to thedisparity of the obstacle. Accordingly, the controller 302 may calculatesimilarity between the disparity of the obstacle boundary and thedisparity of the obstacle. FIGS. 19A-19C are views for describingcalculation of disparity similarity in the form of the presentdisclosure and the result of the calculation. FIG. 19A shows density ofa disparity map, and FIG. 19B shows a calculated result of the disparitysimilarity. In the calculated result of the disparity similarity asshown in FIG. 19B, a position at which a difference between a sum ofsimilarities of vertical direction disparities of an arrow areaindicated by a reference numeral 1902 and a sum of similarities ofvertical direction disparities of an arrow area indicated by a referencenumeral 1904 becomes maximum is a position indicating the height of theobstacle. Therefore, a difference between disparity similarity sums ofareas above and below a specific location on the disparity similaritymay be set to a distance based cost map. FIG. 19C shows a calculatedresult of a distance based cost map in the form of the presentdisclosure.

Returning to operation 406 of FIG. 6, the controller 302 may create acolor based cost map from the converted image, in operation 608. FIG. 20is a view for describing operation of setting an area of interest andmeasuring disparities in order to obtain information about a height ofan obstacle based on color in one form of the present disclosure. InFIG. 20, feature points (portions indicated by circles) participating indetecting an obstacle boundary are likely to be present on the obstacle.Therefore, by measuring color similarities between the feature points,probabilities that the features points belong to the obstacle can bedetermined. As shown in FIG. 20, the controller 302 may set an area forwhich color similarity will be measured to an area 2002 of interest,measure color similarities between all feature points existing in thearea 2002 of interest, and select a greatest color similarity value as afinal color similarity value.

A color similarity may be calculated by subtracting a maximum value froma Euclidean distance of a RGB value. FIGS. 21A-21C are views fordescribing a process of calculating a color-based cost map based oncolor similarity calculation. FIG. 21A shows a converted image andfeature points, FIG. 21B shows the result of color similaritycalculation, and FIG. 21C shows a result of the color based cost mapcalculation. As shown in FIG. 21B, a position at which a greatsimilarity difference appears in the vertical direction of an image is aposition indicating a height of an obstacle. Accordingly, the controller302 may calculate a difference between color similarity sums forpredetermined areas below and above a specific position from the colorsimilarity calculation result, and create a color based cost map asshown in FIG. 21C based on the difference.

Returning to operation 406 of FIG. 6, the controller 302 may create anedge based cost map from the converted image, in operation 610.Generally, there is a strong edge component at a boundary between anobstacle and a non-obstacle. Since the result of edge detection has abinary value, it is inappropriate to use the result of edge detection asa cost map. Therefore, the controller 302 may modify the result of edgedetection result through distance transform such that pixels locatedcloser to edges have lower values, thereby creating an edge based costmap. FIGS. 22A-22B are views for describing calculation of an edge basedcost map based on edge detection in one form of the present disclosure.FIG. 22B shows an edge based cost map calculated based on one form ofthe present disclosure.

Returning to operation 406 of FIG. 6, the controller 302 may combine thedistance based cost map, the color based cost map, and the edge basedcost map to generate an integrated final cost map as shown in the lowerleft figure of FIG. 23, in operation 612.

Also, the controller 302 may estimate the height of the obstacle throughDP-based information fusion from the final cost map, in operation 614.The controller 302 may assign weights to a cost map calculated based ondistance c_(d), color c_(c), and edge c_(e), as expressed by Equation 6below, and then calculate a weighted sum.

c _(t) =w _(d) c _(d) +w _(c) c _(c) +w _(e) c _(e)  <Equation 6>

In Equation 6, w_(d), w_(c), and w_(e) are weights of the distance costmap, the color cost map, and the edge cost map, respectively. c_(t)obtained by the weighted sum may be the cost map for estimating thefinal height of the obstacle. The height of the obstacle may beestimated by applying DP to the final cost map. FIG. 23 shows theresults obtained by applying cost map integration, application of DP,and the results of final height estimation of the obstacle to an actualimage.

FIG. 24 shows final results obtained by estimating the boundary andheight of the obstacle through a series of processes in one form of thepresent disclosure. As shown in FIG. 24, the boundary and height of theobstacle may be accurately detected by the monocular camera 102 throughthe obstacle boundary estimation and the obstacle height estimation inthe forms of the present disclosure.

It is to be understood that the above description is only illustrativeof technical ideas, and various modifications, alterations, andsubstitutions are possible without departing from the desiredcharacteristics of the present disclosure. Therefore, the forms and theaccompanying drawings described above are intended to illustrate and notlimit the technical idea, and the scope of technical thought is notlimited by these forms and accompanying drawings. The scope of which isto be construed in accordance with the present disclosure, and alltechnical ideas which are within the scope of the same should beinterpreted as being included in the scope of the right.

What is claimed is:
 1. A method of detecting an obstacle around avehicle, comprising: acquiring an image of the obstacle around thevehicle using a monocular camera; creating, by a controller, a distancebased cost map, a color based cost map and an edge based cost map fromthe image; and integrating, by the controller, the distance based costmap, the color based cost map, and the edge based cost map to generate afinal cost map, and estimating by the controller a height of theobstacle from the final cost map.
 2. The method of claim 1, wherein thecreating of the distance based cost map comprises: generating aplurality of Delaunay triangles by connecting feature points in theimage and performing interpolation of distance information using a planeequation generated by three vertexes of each of the plurality of theDelaunay triangles; and estimating a disparity of pixels included ineach of the plurality of the Delaunay triangles from a result of theinterpolation.
 3. The method of claim 1, wherein the creating of thecolor based cost map comprises: setting an area where color similaritiesare measured in the image, measuring the color similarities between allfeature points existing in the area, and selecting a greatest colorsimilarity value as a final color similarity; and calculating adifference between sums of the color similarities from the measuredcolor similarities to create the color based cost map.
 4. The method ofclaim 1, wherein the creating of the edge based cost map comprises:performing edge detection on the image; and performing distancetransformation based on edge detection such that pixels located closerto an edge have lower values.
 5. The method of claim 1, wherein thefinal cost map is calculated as:c _(t) =w _(d) c _(d) +w _(c) c _(c) +w _(e) c _(e), where: w_(d) is aweight of the distance cost map, w_(c) is a weight of the color costmap, w_(e) is a weight of the edge cost map, c_(d), color c_(c), andedge c_(e).
 6. A method of detecting an obstacle around a vehicle,comprising: acquiring an image of the obstacle around the vehicle usinga monocular camera; reconstructing, by a controller, three-dimensionalpositions of corresponding points in the image; integrating, by thecontroller, previously reconstructed three-dimensional correspondingpoints with the currently reconstructed three-dimensional correspondingpoints based on a relative positional relationship between thepreviously reconstructed three-dimensional corresponding points and thecurrently reconstructed three-dimensional corresponding points;calculating, by the controller, a disparity value by applying a virtualbaseline value formed by a movement of the monocular camera to a depthvalue obtained through three-dimensional reconstruction of thecorresponding points; and estimating, by the controller, a boundary ofthe obstacle based on the disparity value.
 7. The method of claim 6,wherein the disparity value is calculated as: ${d = {f\frac{B}{Z}}},$where, the d is the disparity value to be obtained, the B is the virtualbaseline value of the monocular camera, the Z is the depth valueobtained through three-dimensional reconstruction, and the f is a focallength of the monocular camera.
 8. The method of claim 6, furthercomprises changing, when an angle of view of the monocular camera is awide angle that is greater than or equal to a predetermined angle ofview, a u-axis to an incident angular axis θu through the followingequation: ${\theta_{u} = {{atan}\left( \frac{u - o_{x}}{f} \right)}},$where, the u is the value of the u-axis, the 0_(x) is the center pointof the monocular camera, and the f is the focal length of the monocularcamera.
 9. A method of detecting an obstacle around a vehicle,comprising: acquiring an image of the obstacle around the vehicle usinga monocular camera; reconstructing, by a controller, three-dimensionalpositions of corresponding points in the image, calculating, by thecontroller, a disparity value by applying a virtual baseline valueformed by a movement of the monocular camera to a depth value obtainedthrough three-dimensional reconstruction of the corresponding points,and estimating a boundary of the obstacle based on the disparity valueby the controller; and creating, by the controller, a distance basedcost map, a color based cost map and an edge based cost map from theimage, and estimating, by the controller, a height of the obstacle usingthe distance based cost map, the color based cost map, and the edgebased cost map.
 10. The method of claim 9, wherein the disparity valueis calculated as: ${d = {f\frac{B}{Z}}},$ where the ‘d’ is thedisparity value to be obtained, the B is the virtual baseline value ofthe monocular camera, the Z is the depth value obtained throughthree-dimensional reconstruction, and the f is a focal length of themonocular camera.
 11. The method of claim 9, further comprisingchanging, when an angle of view of the monocular camera is a wide anglethat is greater than or equal to a predetermined angle of view, a u-axisto an incident angular axis θu through the following equation:${\theta_{u} = {{atan}\left( \frac{u - o_{x}}{f} \right)}},$ where the uis the value of the u-axis, the 0_(x) is the center point of themonocular camera, and the f is the focal length of the monocular camera.12. The method of claim 9, wherein the creating of the distance basedcost map comprises: generating a plurality of Delaunay triangles byconnecting feature points of the image and performing interpolation ofdistance information using a plane equation generated by three vertexesof each of the plurality of the Delaunay triangles; and estimating adisparity of pixels included in each of the plurality of the Delaunaytriangles from a result of the interpolation.
 13. The method of claim 9,wherein the creating of the color based cost map comprises: setting anarea where color similarity measurement is to be performed in the image,measuring color similarities between all feature points existing in thearea, and selecting a greatest color similarity value as a final colorsimilarity; and creating the color based cost map by calculating adifference between sums of the color similarities from the measuredcolor similarities.
 14. The method of claim 9, wherein the creating ofthe edge based cost map comprises: performing edge detection on theimage; and performing distance transform on a result of the edgedetection such that pixels located closer to edges have lower values.15. The method of claim 9, wherein the final cost map is calculated as:c _(t) =w _(d) c _(d) +w _(c) c _(c) +w _(e) c _(e), where w_(d) is aweight of the distance cost map, w_(c) is a weight of the color costmap, w_(e) is a weight of the edge cost map, and c_(d), color c_(c), andedge c_(e).
 16. A method of detecting an obstacle around a vehicle,comprising: acquiring an image of the obstacle around the vehicle usinga monocular camera; and reconstructing, by a controller,three-dimensional positions of corresponding points in the image,calculating by the controoler a disparity value by applying a virtualbaseline value formed by a movement of the monocular camera to a depthvalue obtained through the three-dimensional reconstruction of thecorresponding points, and estimating a boundary of the obstacle based onthe disparity value by the controller.
 17. A method of detecting anobstacle around a vehicle, comprising: acquiring an image of theobstacle around the vehicle using a monocular camera; and creating, by acontroller, a distance based cost map, a color based cost map, and anedge based cost map from the image, and estimating by the controller aheight of the obstacle using the distance based cost map, the colorbased cost map, and the edge based cost map.